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Digital First Assessment Note 


MPBSD: ServiceOntario Driver and Vehicle 
Conversational IVR (Phase 1 - MVP) 


Date of assessment (DA-690): November 15, 2022 


Service design phase being assessed: mid-Beta 


July 8, 2021, November 30, Not assessed November 15, 

2021 2021 2022 approved 

approved course correct met 13/13 points 
met 9/13 points 


Executive summary 


The product 


The ServiceOntario Contact Centre receives calls from the public on a number of programs 
including Driver and Vehicle (D&V), which represents a significant portion of the calls 
(approximately 40-45%). D&V is a complex program and the contact centre must field a diverse 
array of calls. This has become difficult to manage within a traditional interactive voice response 
(IVR) solution. The team is aware through analysis of available data that approximately 25% of 
callers don't sit through the current IVR and skip right to speak to an agent which leads to long wait 
times. This project seeks to implement a Conversational IVR (CIVR) so callers will no longer select 
menu options 1-5 on their phones or navigate multiple layers of menu options and instead they will 
say what they are looking for and the conversational IVR will determine the best way to respond. 


Since receiving a Course Correct at their last DFA, the team has connected with the ODS UX and 
content teams and made considerable progress across all the recommendations. 


As the service moves on from Beta it is transitioning from Digital Enterprise Engagement Practice 
(DEEP) within the Government Services Integration Cluster (GSIC) to ownership by Service Ontario. 


Additional info: 


e Users: Ontario residents who drive and/or own a vehicle; organizations who own vehicles 
used by employees; companies that rent vehicles to individuals; insurance companies; 
enforcement officers; newcomers and visitors to Ontario. 

e Cost: 1. 45M - 662K for resources and 790K for technology. Funding has been approved as 
part of MTO’s Digital First Treasury Board submission. 

e Timeline: Target launch November 28th, 2022. 


The team 


The assessment was attended by: 

e Delivery team: Kim Monastyrskyj, Beth Preszcator, Jerome Song, Tim Trudeau, Carolyn 
Luney, Kyle Kwan, Gleb Khoroujik, Matthew Hayvren, Diwakar V. Singh, Malaika Christie, Jerry 
Hamilton 

e Assessors team: Samantha Sharkov, Adie Margineau, Alireza Tayari, Devon Codrington, 
Observer: Vatsal Shah 

e DFA team: Keith Robinson, Rayna Taylor, Melissa Johnston 
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Digital First Assessment result: Approved 


Digital First Assessments are a consistent way to evaluate the design, build and delivery of digital 
products and services against Ontario's Digital Service Standard. 


There are 3 possible outcomes: Approved, Course correct, and Halt. 


The design and development of this digital service is in alignment with 13 of the 13 points of the 
Digital Service Standard. The result of this assessment is approved at the mid-Beta phase of the 
project. 


Key recommendations 


The assessors noted the following priorities for this initiative. We understand that some of the 
identified priorities may seem out of scope for the core delivery team and may need executive level 
support. These have been sectioned accordingly. 


For delivery team 


As you move forward into the next phase of service design: 


1. The service has not yet been tested with users with disability needs. While it’s good to hear a 
plan to do so is being developed, it’s imperative that the testing happens and the feedback is 
incorporated before going live. 

2. The team has done a great job setting KPIs, measuring results and making changes based on 
the data. The team has set goals of improving on baseline data by 10 %. As a next step 
consider more rigour around setting targets. 

3. While the service tests well, the assessors noted some potential usability issues (identified 
below under ‘Ensure users succeed the first time,’ that the team can address as they move 
ahead. 


For executive leadership 


1. With ownership transferring after launch there is always some risk that the hard work of the 
delivery team will not be adequately maintained and extended. The team has a good 
sustainment plan, but it's important that ServiceOntario continue to further develop its 
digital capacity as is outlined in the submission. 
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Assessment by DSS point 


The following notes are based on the feedback that was provided by the Assessor team after 
reviewing the submission and the follow up discussion at the Assessment meeting. It includes 
advice for the team to consider moving forward. 


1.Understand users and their needs 
Result: Met the standard 


Summary findings 


e The team demonstrated that they followed the advice from the last DFA assessment, 
working closely with Service Ontario to implement recommendations in terms of creating 
consistent content across all platforms, designing a service from start to finish and 
measuring performance all after consulting with the ODS and conducting deeper user 
research with actual participants. 

e The solution is based on DEEP’s Enterprise Platform (EP) .The project team has worked 
collaboratively to create a “reusable” application. CIVR can be leveraged by other clients. The 
project team has created a framework that can be followed by future project teams. 


What the team has done well 


The team had a strong problem statement and informative user profiles. 
The team expressed that the user research has truly helped them better understand user 
needs, motivations and frustrations and led to substantial improvements in the service. 

e The team is co-ordinating well with other services in same space: i.e. Live Chat, Chat bot, and 
Service Ontario centres to develop consistent content. 


What the team needs to explore going forward 
e The team should continue to test the service with users and make improvements based on 
feedback. 


e Theteam should continue to leverage learnings and co-ordinate with similar Service Ontario 
services. 


2. Establish the right team 
Result: Met the standard 


Summary findings 


e The extended team includes GSIC, in collaboration with ServiceOntario, Contact centres, 
communications, policy, ODS research, measurement and analysis, MTO policy and partner 
services. 

e The core team includes a product manager, user research, content design and service design 
roles. 

e ServiceOntario Customer Care is the business owner and will manage the content, 
translation and approvals, working in partnership with DEEP who support technology 
-backend in this initial phase of the pilot. 

e DEEP can provide project support for - CIVR when it moves forward, Chatbot and CIVR when 
it is ready to expand and move beyond pilot. 

e As ServiceOntario further develops its digital capacity, additional staff with the digital and 
content expertise along with the business expertise will be added to the team. 


What the team has done well 


e Great to see product manager, content design and UX roles on the team. 
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What the team needs to explore going forward 


e With ownership transfer after launch there is always some risk that the hard work of the 
delivery team will not be adequately maintained. The team has a good sustainment plan, but 
it's important that ServiceOntario further develop its digital capacity as is outlined in the 
submission. 


3. Be consistent 
Result: Met the standard 


Summary findings 


e Asavoice activated solution, with no change to the phone number, the service does not 
have a web component and as such the Ontario Design System is not relevant. 

e Since the last assessment the team has worked to achieve consistency across the various 
Service Ontario channels - centres, Chat bot, web etc. 

e The team has followed Ontario.ca content standards. 


What the team has done well 


e Great to see the team making strides to achieve cross platform consistency and improve the 
quality of the content. 


What the team needs to explore going forward 


e The team should continue to review information with stakeholders - contact centres, MTO, 
and application teams to ensure consistency across all channels. 


4. Design the service from start to finish 
Result: Met the standard 


Summary findings 


e In fall of 2020 DEEP conducted a lean assessment with ServiceOntario and presented 
immediate, medium and long-term solutions to the Customer Care ADM for approval. 
The team developed end-to-end customer journey maps as part of the service design. 
The team is soliciting ongoing feedback via surveys aswell as by analysing data. 
The team has demonstrated how conversational IVR is one of several service channels that 
meet different Ontarians needs. 


What the team has done well 


e The team has acknowledged that designing the service from start to finish is essential for a 
medium like IVR. 


What the team needs to explore going forward 


e Continue to conduct user research as the service evolves in order to understand the user 
journey. 


5. Ensure users succeed the first time 
Result: Met the standard 


Summary findings 


e Content and flow were based on user experience, workshops, and research and was tested 
with users. 

e The team conducts daily reviews of chatbot and CIVR analytics including number of calls, 
what the caller said (missed and understood), and transfers to agents. 
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What the team has done well 


e The system collects alot of detailed user data and the team has a process for analysing it 
and using this to improve the service. 


What the team needs to explore going forward 


e In our walk through of the service the assessors noticed some possible usability concerns 
that the team can investigate: 

o There should be a clear statement that this is an “automated service and you're talking 
to a virtual assistant”. 

o After hearing the customer's issue, a best practice is to repeat their question / 
rephrase the question to ensure that we are understanding them properly. 

o If yes, mentioning our “willingness” to help. Then providing them the answer. If not, try 
to ask for clarification. On the second (or third) failure, porting them to a real agent is 
a good best practice. 


6. Test the end to end service 
Result: Met the standard 


Summary findings 
DEEP platform is an enterprise offering and it is a scalable solution to engage the public. 
Dedicated DEEP team tests all new changes and completes regression testing and 
application functionality. 
e Business testing is ongoing. 


What the team needs to explore going forward 


e Asthe team mentioned, this isn't a ‘one and done’ scenario but rather requires continuous 
testing. 


7. Make it accessible and inclusive 
Result: Met the standard 


Summary findings 

e Based on research, examples of groups who may experience accessibility barriers include: 
individuals with hearing impairment; Individuals with cognitive disabilities and Individuals 
with speech and language impairments. These groups have been considered during the 
design of the solution, and accessibility accommodation will be incorporated in the call flow 
design. 

e Targeted outreach to people with disabilities has not been included at this time. At the 
recommendation of the ODS research team this action can be taken later in the evolution of 
the service channel. ODS is currently working on a recruitment strategy related to 
accessibility. 

e People with hearing impairment can use the TTY line. Users who have difficulty vocalising in 
a way the system understands can speak to an agent. 


What the team needs to explore going forward 


It's imperative that the team test the service with a diverse set of users with disability needs. 
The team should look at their existing data and call logs to identify any trends or insights 
related to difficulties experienced by people with disabilities (can be inferred from 
speech/missed utterances, deferring to TTY, deferring to agent). 

e Consider doing a limited pilot asking callers to disclose if they use any accessibility 
accommodations in their daily life and reporting on call experience for individuals that use 
accommodations. 

e Reach out to the ACOE team for further advice. 
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8. Be agile and user-centred 
Result: Met the standard 


Summary findings 
e SWAT meetings and an asynchronous Kanban board have been effective in creating updates 
for MTO, Drive Test, MPBSD, ServiceOntario, Web, Social Media, Applications, and Contact 
Centre. 
e An agile approval process has been developed and improved. It will continue to evolve with 
ServiceOntario transformation. 


What the team has done well 


e The team maintains weekly stand-ups, works in sprints, and maintains a backlog. 


What the team needs to explore going forward 


e The team seems to have good agile procedures in place. Again what is important is 
maintaining and growing these ways of working as things transition to sustainment. 


9. Use open standards and common platforms 
Result: Met the standard 


Summary findings 


AWS’ cloud based natural language Al technology is used to process the chat conversation. 
DEEP’s Engagement Platform used to host the call queues, which is integrated with the CIVR. 
This MVP will be the first of its kind in the Ontario government, and will provide learnings 
that can be leveraged for future CIVR implementations within the OPS. 

e No open-source options were used. 


10. Embed privacy and security by design 
Result: Met the standard 


Summary findings 


e The solution will not store, collect, use, or disclose sensitive personal information. The 
project team has followed Privacy recommendations from the MPBSD Privacy Office relating 
to minimizing any risk for inadvertently collecting, using, storing personal information. 

e Corporate Security; Information, Privacy and Archives; Cyber security; and Internal Audit, 
were all engaged. A TRA was completed. 

e The Head of Privacy for MPBSD stated that no PIA was required. 


11. Support those who need it 
Result: Met the standard 


Summary findings 


In a traditional touch tone IVR, the caller is required to ‘listen’ and press buttons. With a 
Conversational IVR, they will be asked to speak their intentions or ask their questions. The design 
should allow for scenarios where the caller may require accommodations to perform these actions. 
The team has adopted the following strategies to assist users who require support: 


e TTY (Teletype) - callers who have their own TTY device can call into ECCS - when the IVR 
receives TTY calls, they are transferred to another phone number received and managed by 
the OPS Enterprise TTY service - where an agent can communicate with the caller through 
text using the OPS TTY software installed on their desktop. 
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e Users can switch to a live agent by saying “Agent” or pressing zero, bothavailable in the 
Conversational IVR design. 


What the team needs to explore going forward 


e As mentioned earlier, the team should make it more clear to users how to transfer to 
a live agent. 


12. Measure performance 
Result: Met the standard 


Summary findings 


e The team has set relevant KPIs. 
e The team has access to great data including call volume numbers, number of calls received, 
busy signals, calls that went to an agent, responses that were not recognized etc. 


What the team has done well 


e The team has set relevant KPIs, has the capability to measure them and a track record of 
making improvements based on the data. 


What the team needs to explore going forward 


e The team has set goals of improving on baseline measurements by 10 %. While it’s great to 
see success criteria, the team might want to apply more rigour in setting targets. 

e Use the Digital Performance Measurement resources to help identify KPIs for the service and 
overall outcomes. 


13. Be a good data steward 
Result: Met the standard 


Summary findings 


Currently, ServiceOntario wait times and call volumes (contact centres) are published to the 
Ontario data catalogue: ServiceOntario wait times and call volumes (contact centres) - Datasets - 
Ontario Data Catalogue. This includes D&V data. This data will continue to be published. 


Next Assessment 


The next assessment for this project is due at after-Live. We are currently implementing a new 
process for after-Live assessments and will connect with you directly to initiate this process. 


Have Questions? 


Our goal is to provide clear, consistent and usable advice around the application of the Digital 
Service Standard and digital best practices. If you run into any problems or need clarity about any 
of the advice provided we are always available to talk to you and your team. Please reach out to 


digitalassessments@ontario.ca if you have any questions. 


